Denial-Artificial Intelligence Tools and Health Insurance Coverage Decisions.

This JAMA Forum discusses the potential and the pitfalls in the use of artificial intelligence in the coverage decisions made by health insurance companies.

Bias-Adjusted Predictions of County-Level Vaccination Coverage from the COVID-19 Trends and Impact Survey.

BACKGROUND: The potential for selection bias in nonrepresentative, large-scale, low-cost survey data can limit their utility for population health measurement and public health decision making. We developed an approach to bias adjust county-level COVID-19 vaccination coverage predictions from the large-scale US COVID-19 Trends and Impact Survey. DESIGN: We developed a multistep regression framework to adjust for selection bias in predicted county-level vaccination coverage plateaus. Our approach included poststratification to the American Community Survey, adjusting for differences in observed covariates, and secondary normalization to an unbiased reference indicator. As a case study, we prospectively applied this framework to predict county-level long-run vaccination coverage among children ages 5 to 11 y. We evaluated our approach against an interim observed measure of 3-mo coverage for children ages 5 to 11 y and used long-term coverage estimates to monitor equity in the pace of vaccination scale up. RESULTS: Our predictions suggested a low ceiling on long-term national vaccination coverage (46%), detected substantial geographic heterogeneity (ranging from 11% to 91% across counties in the United States), and highlighted widespread disparities in the pace of scale up in the 3 mo following Emergency Use Authorization of COVID-19 vaccination for 5- to 11-y-olds. LIMITATIONS: We relied on historical relationships between vaccination hesitancy and observed coverage, which may not capture rapid changes in the COVID-19 policy and epidemiologic landscape. CONCLUSIONS: Our analysis demonstrates an approach to leverage differing strengths of multiple sources of information to produce estimates on the time scale and geographic scale necessary for proactive decision making. IMPLICATIONS: Designing integrated health measurement systems that combine sources with different advantages across the spectrum of timeliness, spatial resolution, and representativeness can maximize the benefits of data collection relative to costs. HIGHLIGHTS: The COVID-19 pandemic catalyzed massive survey data collection efforts that prioritized timeliness and sample size over population representativeness.The potential for selection bias in these large-scale, low-cost, nonrepresentative data has led to questions about their utility for population health measurement.We developed a multistep regression framework to bias adjust county-level vaccination coverage predictions from the largest public health survey conducted in the United States to date: the US COVID-19 Trends and Impact Survey.Our study demonstrates the value of leveraging differing strengths of multiple data sources to generate estimates on the time scale and geographic scale necessary for proactive public health decision making.

A call for better validation of opioid overdose risk algorithms.

Clinical decision support (CDS) systems powered by predictive models have the potential to improve the accuracy and efficiency of clinical decision-making. However, without sufficient validation, these systems have the potential to mislead clinicians and harm patients. This is especially true for CDS systems used by opioid prescribers and dispensers, where a flawed prediction can directly harm patients. To prevent these harms, regulators and researchers have proposed guidance for validating predictive models and CDS systems. However, this guidance is not universally followed and is not required by law. We call on CDS developers, deployers, and users to hold these systems to higher standards of clinical and technical validation. We provide a case study on two CDS systems deployed on a national scale in the United States for predicting a patient's risk of adverse opioid-related events: the Stratification Tool for Opioid Risk Mitigation (STORM), used by the Veterans Health Administration, and NarxCare, a commercial system.

Conditional cross-design synthesis estimators for generalizability in Medicaid.

While much of the causal inference literature has focused on addressing internal validity biases, both internal and external validity are necessary for unbiased estimates in a target population of interest. However, few generalizability approaches exist for estimating causal quantities in a target population that is not well-represented by a randomized study but is reflected when additionally incorporating observational data. To generalize to a target population represented by a union of these data, we propose a novel class of conditional cross-design synthesis estimators that combine randomized and observational data, while addressing their estimates' respective biases-lack of overlap and unmeasured confounding. These methods enable estimating the causal effect of managed care plans on health care spending among Medicaid beneficiaries in New York City, which requires obtaining estimates for the 7% of beneficiaries randomized to a plan and 93% who choose a plan, who do not resemble randomized beneficiaries. Our new estimators include outcome regression, propensity weighting, and double robust approaches. All use the covariate overlap between the randomized and observational data to remove potential unmeasured confounding bias. Applying these methods, we find substantial heterogeneity in spending effects across managed care plans. This has major implications for our understanding of Medicaid, where this heterogeneity has previously been hidden. Additionally, we demonstrate that unmeasured confounding rather than lack of overlap poses a larger concern in this setting.

Long-Term Prospects for Telemedicine in Opioid Use Disorder (OUD) Treatment: Results from a Longitudinal Survey of OUD Clinicians.

BACKGROUND: During the pandemic, there was a dramatic shift to telemedicine for opioid use disorder (OUD) treatment. Little is known about how clinician attitudes about telemedicine use for OUD treatment are evolving or their preferences for future use. OBJECTIVE: To understand OUD clinician views of and preferences regarding telemedicine. DESIGN: Longitudinal survey (wave 1, December 2020; wave 2, March 2022). SUBJECTS: National sample of 425 clinicians who treat OUD. MAIN MEASURES: Self-reported proportion of OUD visits delivered via telemedicine (actual vs. preferred), comfort in using video visits for OUD, impact of telemedicine on work-related well-being. KEY RESULTS: The mean reported percentage of OUD visits delivered via telemedicine (vs. in person) dropped from 56.9% in December 2020 to 41.5% in March 2022; the mean preferred post-pandemic percentage of OUD visits delivered via telemedicine was 34.8%. Responses about comfort in using video visits for different types of OUD patients remained similar over time despite clinicians having substantially more experience with telemedicine by spring 2022 (e.g., 35.8% vs. 36.0% report being comfortable using video visits for new patients). Almost three-quarters (70.9%) reported that most of their patients preferred to have the majority of their visits via telemedicine, and 76.7% agreed that the option to do video visits helped their patients remain in treatment longer. The majority (58.7%) reported that telemedicine had a positive impact on their work-related well-being, with higher rates of a positive impact among those who completed training more recently (68.5% of those with < 10 years, 62.1% with 10-19 years, and 45.8% with 20 + years, p < 0.001). CONCLUSIONS: While many surveyed OUD clinicians were not comfortable using telemedicine for all types of patients, most wanted telemedicine to account for a substantial fraction of OUD visits, and most believed telemedicine has had positive impacts for themselves and their patients.

Many Clinicians Implement Digital Equity Strategies To Treat Opioid Use Disorder.

Drawing upon a longitudinal survey of clinicians who treat patients with opioid use disorder (OUD), we report changes over time in telemedicine use, clinicians' attitudes, and digital equity strategies. Clinicians reported less use of telemedicine (both video and audio-only) in 2022 than in 2020. In March 2022, 77.0 percent of clinician respondents reported implementing digital equity strategies to help patients overcome barriers to video visits.

Measuring concordance of data sources used for infectious disease research in the USA: a retrospective data analysis.

OBJECTIVES: As highlighted by the COVID-19 pandemic, researchers are eager to make use of a wide variety of data sources, both government-sponsored and alternative, to characterise the epidemiology of infectious diseases. The objective of this study is to investigate the strengths and limitations of sources currently being used for research. DESIGN: Retrospective descriptive analysis. PRIMARY AND SECONDARY OUTCOME MEASURES: Yearly number of national-level and state-level disease-specific case counts and disease clusters for three diseases (measles, mumps and varicella) during a 5-year study period (2013-2017) across four different data sources: Optum (health insurance billing claims data), HealthMap (online news surveillance data), Morbidity and Mortality Weekly Reports (official government reports) and National Notifiable Disease Surveillance System (government case surveillance data). RESULTS: Our study demonstrated drastic differences in reported infectious disease incidence across data sources. When compared with the other three sources of interest, Optum data showed substantially higher, implausible standardised case counts for all three diseases. Although there was some concordance in identified state-level case counts and disease clusters, all four sources identified variations in state-level reporting. CONCLUSIONS: Researchers should consider data source limitations when attempting to characterise the epidemiology of infectious diseases. Some data sources, such as billing claims data, may be unsuitable for epidemiological research within the infectious disease context.

A cohort study examining changes in treatment patterns for alcohol use disorder among commercially insured adults in the United States during the COVID-19 pandemic.

INTRODUCTION: We know very little about how the pandemic impacted outpatient alcohol use disorder (AUD) care and the role of telemedicine. METHODS: Using OptumLabs® Data Warehouse de-identified administrative claims, we identified AUD cohorts in 2018 (N = 23,204) and 2019 (N = 23,445) and examined outpatient visits the following year, focusing on week 12, corresponding to the March 2020 US COVID-19 emergency declaration, through week 52. Using multivariable logistic regression, we examined the association between patient demographic and clinical characteristics and receipt of any outpatient AUD visits in 2020 vs. 2019. RESULTS: In 2020, weekly AUD visit utilization decreased maximally at the pandemic start (week 12) by 22.5 % (2019: 3.8 %, 2020: 3.0 %, percentage point change [95 % CI] = -0.86[-1.19, -0.05]) but was similar to 2019 utilization by mid-April 2020 (week 16). Telemedicine accounted for 50.1 % of AUD visits by early July 2020 (week 27). Individual therapy returned to 2019 levels within 1 week (i.e., week 13) whereas group therapy did not consistently do so until mid-August 2020 (week 31). Further, individual therapy exceeded 2019 levels by as much as 50 % starting mid-October 2020. The study found no substantial differences in visits by patient demographic or clinical characteristics. CONCLUSIONS: Among patients with known AUD, initial outpatient care disruptions were relatively brief. However, substantial shifts occurred in care delivery-an embrace of telemedicine but also more pronounced, longer disruptions in group therapy vs. individual and an increase in individual therapy use. Further research needs to help us understand the implications of these findings for clinical outcomes.

Uncertainty in lung cancer stage for survival estimation via set-valued classification.

The difficulty in identifying cancer stage in health care claims data has limited oncology quality of care and health outcomes research. We fit prediction algorithms for classifying lung cancer stage into three classes (stages I/II, stage III, and stage IV) using claims data, and then demonstrate a method for incorporating the classification uncertainty in survival estimation. Leveraging set-valued classification and split conformal inference, we show how a fixed algorithm developed in one cohort of data may be deployed in another, while rigorously accounting for uncertainty from the initial classification step. We demonstrate this process using SEER cancer registry data linked with Medicare claims data.

Data Source Concordance for Infectious Disease Epidemiology.

Background: As highlighted by the COVID-19 pandemic, researchers are eager to make use of a wide variety of data sources, both government-sponsored and alternative, to characterize the epidemiology of infectious diseases. To date, few studies have investigated the strengths and limitations of sources currently being used for such research. These are critical for policy makers to understand when interpreting study findings. Methods: To fill this gap in the literature, we compared infectious disease reporting for three diseases (measles, mumps, and varicella) across four different data sources: Optum (health insurance billing claims data), HealthMap (online news surveillance data), Morbidity and Mortality Weekly Reports (official government reports), and National Notifiable Disease Surveillance System (government case surveillance data). We reported the yearly number of national- and state-level disease-specific case counts and disease clusters according to each of our sources during a five-year study period (2013-2017). Findings: Our study demonstrated drastic differences in reported infectious disease incidence across data sources. When compared against the other three sources of interest, Optum data showed substantially higher, implausible standardized case counts for all three diseases. Although there was some concordance in identified state-level case counts and disease clusters, all four sources identified variations in state-level reporting. Interpretation: Researchers should consider data source limitations when attempting to characterize the epidemiology of infectious diseases. Some data sources, such as billing claims data, may be unsuitable for epidemiological research within the infectious disease context.

Association Between Telemedicine Use in Nonmetropolitan Counties and Quality of Care Received by Medicare Beneficiaries With Serious Mental Illness.

Importance: Access to specialty mental health care remains challenging for people with serious mental illnesses, such as schizophrenia and bipolar disorder. Whether expansion of telemedicine is associated with improved access and quality of care for these patients is unclear. Objective: To assess whether greater telemedicine use in a nonmetropolitan county is associated with quality measures, including use of specialty mental health care and medication adherence. Design, Setting, and Participants: In this cohort study, the variable uptake of telemental health visits was examined across a national sample of fee-for-service claims from Medicare beneficiaries in 2916 nonmetropolitan counties between January 1, 2010, and December 31, 2018. Beneficiaries with schizophrenia and related psychotic disorders and/or bipolar I disorder during the study period were included. For each year of the study, each county was categorized based on per capita telemental health service use (none, low, moderate, and high). The association between telemental health service use in the county and quality measures was tested using a multivariate model controlling for both patient characteristics and county fixed effects. Analyses were conducted from January 1 to April 11, 2022. Before the COVID-19 pandemic, telemedicine reimbursement was limited to nonmetropolitan beneficiaries. Main Outcomes and Measures: Receipt of a minimum of 2 specialty mental health service visits (telemedicine or in-person) in the year, number of months per year with medication, hospitalization rate, and outpatient follow-up visits after a mental health hospitalization in a year. Results: In 2018, there were 2916 counties with 118 170 patients (77 068 [65.2%] men; mean [SD] age, 58.3 [15.6] years) in the sample. The fraction of counties that had high telemental health service use increased from 2% in 2010 to 17% in 2018. In 2018 there were 1.08 telemental health service visits per patient in the high telemental health counties. Compared with no telemental health care in the county, patients in high-use counties were 1.2 percentage points (95% CI, 0.81-1.60 percentage points) (8.0% relative increase) more likely to have a minimum number of specialty mental health service visits, 13.7 percentage points (95% CI, 5.1-22.3 percentage points) (6.5% relative increase) more likely to have outpatient follow-up within 7 days of a mental health hospitalization, and 0.47 percentage points (95% CI, 0.25-0.69 percentage points) (7.6% relative increase) more likely to be hospitalized in a year. Telemental health service use was not associated with changes in medication adherence. Conclusions and Relevance: The findings of this study suggest that greater use of telemental health visits in a county was associated with modest increases in contact with outpatient specialty mental health care professionals and greater likelihood of follow-up after hospitalization. No substantive changes in medication adherence were noted and an increase in mental health hospitalizations occurred.

Disruptions in Care for Medicare Beneficiaries With Severe Mental Illness During the COVID-19 Pandemic.

Importance: Little is known about changes in care for individuals with severe mental illness during the COVID-19 pandemic. Objective: To examine changes in mental health care during the pandemic and the use of telemedicine in outpatient care among Medicare beneficiaries with severe mental illness. Design, Setting, and Participants: This population-based cohort study included Medicare beneficiaries (age ≥18 years) diagnosed with schizophrenia and schizophrenia-related disorders or bipolar I disorder. Care patterns during January to September 2020 for a cohort defined in 2019 were compared with those during January to September 2019 for a cohort defined in 2018. Exposures: Start of COVID-19 pandemic in the United States, defined as week 12 of 2020. Main Outcomes and Measures: Use of mental health-related outpatient visits, emergency department visits, inpatient care, and oral prescription fills for antipsychotics and mood stabilizers during 4-week intervals. Multivariable logistic regression analyses examined whether the pandemic was associated with differential changes in outpatient care across patient characteristics. Results: The 2019 cohort of 686 214 individuals included 389 245 (53.8%) women, 114 073 (15.8%) Black and 526 301 (72.8%) White individuals, and 477 353 individuals (66.0%) younger than 65 years; the 2020 cohort of 723 045 individuals included 367 140 (53.5%) women, 106 699 (15.6%) Black and 497 885 (72.6%) White individuals, and 442 645 individuals (64.5%) younger than 65 years. Compared with 2019, there were large decreases during the pandemic's first month (calendar weeks 12-15) in individuals with outpatient visits (265 169 [36.7%] vs 200 590 [29.2%]; 20.3% decrease), with antipsychotic and mood stabilizer medication prescription fills (216 468 [29.9%] vs 163 796 [23.9%]; 20.3% decrease), with emergency department visits (12 383 [1.7%] vs 8503 [1.2%]; 27.7% decrease), and with hospital admissions (11 564 [1.6%] vs 7912 [1.2%]; 27.9% decrease). By weeks 32 to 35 of 2020, utilization rebounded but remained lower than in 2019, ranging from a relative decrease of 2.5% (outpatient visits) to 12.9% (admissions). During the full pandemic period (weeks 12-39) in 2020, 1 556 403 of 2 743 553 outpatient visits (56.7%) were provided via telemedicine. In multivariable analyses, outpatient visit use during weeks 12 to 25 of 2020 was lower among those with disability (odds ratio, 0.95; 95% CI, 0.93-0.96), and during weeks 26 to 39 of 2020, it was lower among Black vs non-Hispanic White individuals (OR, 0.97; 95% CI, 0.95-0.99) and those with dual Medicaid eligibility (OR, 0.96; 95% CI, 0.95-0.98). Conclusions and Relevance: In this cohort study, despite greater use of telemedicine, individuals with severe mental illness experienced large disruptions in care early in the pandemic. These narrowed but persisted through September 2020. Disruptions were greater for several disadvantaged populations.

Advancing primary care with Artificial Intelligence and Machine Learning.

Primary care is the largest healthcare delivery platform in the US. Facing the Artificial Intelligence and Machine Learning technology (AI/ML) revolution, the primary care community would benefit from a roadmap revealing priority areas and opportunities for developing and integrating AI/ML-driven clinical tools. This article presents a framework that identifies five domains for AI/ML integration in primary care to support care delivery transformation and achieve the Quintuple Aims of the healthcare system. We concluded that primary care plays a critical role in developing, introducing, implementing, and monitoring AI/ML tools in healthcare and must not be overlooked as AI/ML transforms healthcare.

Improving the Performance of Risk Adjustment Systems: Constrained Regressions, Reinsurance, and Variable Selection.

Modifications of risk-adjustment systems used to pay health plans in individual health insurance markets typically seek to reduce selection incentives at the individual and group levels by adding variables to the payment formula. Adding variables can be costly and lead to unintended incentives for upcoding or service utilization. While these drawbacks are recognized, they are hard to quantify and difficult to balance against the concrete, measurable improvements in fit that may be achieved by adding variables to the formula. This paper takes a different approach to improving the performance of health plan payment systems. Using the HHS-HHC V0519 model from the Marketplaces as a starting point, we constrain fit at the individual and group level to be as good or better than the current payment model while reducing the number of variables in the model. We introduce three elements in the design of plan payment: reinsurance, constrained regressions, and machine learning methods for variable selection. The fit performance of our alternative formulas with many fewer variables is as good or better than the current HHS-HHC V0519 formula.

Considerations Across Three Cultures: Parametric Regressions, Interpretable Algorithms, and Complex Algorithms.

We consider an extension of Leo Breiman's thesis from "Statistical Modeling: The Two Cultures" to include a bifurcation of algorithmic modeling, focusing on parametric regressions, interpretable algorithms, and complex (possibly explainable) algorithms.

Revisiting performance metrics for prediction with rare outcomes.

Machine learning algorithms are increasingly used in the clinical literature, claiming advantages over logistic regression. However, they are generally designed to maximize the area under the receiver operating characteristic curve. While area under the receiver operating characteristic curve and other measures of accuracy are commonly reported for evaluating binary prediction problems, these metrics can be misleading. We aim to give clinical and machine learning researchers a realistic medical example of the dangers of relying on a single measure of discriminatory performance to evaluate binary prediction questions. Prediction of medical complications after surgery is a frequent but challenging task because many post-surgery outcomes are rare. We predicted post-surgery mortality among patients in a clinical registry who received at least one aortic valve replacement. Estimation incorporated multiple evaluation metrics and algorithms typically regarded as performing well with rare outcomes, as well as an ensemble and a new extension of the lasso for multiple unordered treatments. Results demonstrated high accuracy for all algorithms with moderate measures of cross-validated area under the receiver operating characteristic curve. False positive rates were <1%, however, true positive rates were <7%, even when paired with a 100% positive predictive value, and graphical representations of calibration were poor. Similar results were seen in simulations, with the addition of high area under the receiver operating characteristic curve (>90%) accompanying low true positive rates. Clinical studies should not primarily report only area under the receiver operating characteristic curve or accuracy.

Identifying undercompensated groups defined by multiple attributes in risk adjustment.

OBJECTIVE: To identify undercompensated groups in plan payment risk adjustment that are defined by multiple attributes with a systematic new approach, improving on the arbitrary and inconsistent nature of existing evaluations. METHODS: Extending the concept of variable importance for single attributes, we construct a measure of 'group importance' in the random forests algorithm to identify groups with multiple attributes that are undercompensated by current risk adjustment formulas. Using 2016-2018 IBM MarketScan and 2015-2018 Medicare claims and enrolment data, we evaluate two risk adjustment scenarios: the risk adjustment formula used in the individual health insurance Marketplaces and the risk adjustment formula used in Medicare. RESULTS: A number of previously unidentified groups with multiple chronic conditions are undercompensated in the Marketplaces risk adjustment formula, while groups without chronic conditions tend to be overcompensated in the Marketplaces. The magnitude of undercompensation when defining groups with multiple attributes is many times larger than with single attributes. No complex groups were found to be consistently undercompensated or overcompensated in the Medicare risk adjustment formula. CONCLUSIONS: Our method is effective at identifying complex undercompensated groups in health plan payment risk adjustment where undercompensation creates incentives for insurers to discriminate against these groups. This work provides policy-makers with new information on potential targets of discrimination in the healthcare system and a path towards more equitable health coverage.

Ethical Machine Learning in Healthcare.

The use of machine learning (ML) in healthcare raises numerous ethical concerns, especially as models can amplify existing health inequities. Here, we outline ethical considerations for equitable ML in the advancement of healthcare. Specifically, we frame ethics of ML in healthcare through the lens of social justice. We describe ongoing efforts and outline challenges in a proposed pipeline of ethical ML in health, ranging from problem selection to postdeployment considerations. We close by summarizing recommendations to address these challenges.